Discriminability under Insensitivity: Joint Time-Frequency Scattering vs Time Scattering

1. Frequency-Dependent Time-Shift (FDTS) discriminability: JTFS is discriminative, while TS isn't.
2. Frequency-shift insensitivity: TS is sensitive, while JTFS optionally isn't.
3. Time-shift insensitivity: both are insensitive, and both can be invariant.

Introduction

FDTS discriminability

• n_partials: total number of partials, or windowed sines.
• partials_f_sep: f_next = f_prev * partials_f_sep. A constant shift in log-frequency is a multiplication.
• seg_len: partials' effective temporal support.
• f0: initial partial's frequency (which is also the lowest).
• total_shift: total time shift, in number of samples.

• T: should be set to signal's duration, i.e. global averaging, to maximize TS's insensitivity to FDTS.
• Q & r_psi: chosen to yield good time-frequency resolution tradeoff for the example, and sufficient discretization (see r_psi docs) for faithful displayed plot quality.
• pad_mode: zero-padding is best suited for shift-related illustrations. (E.g. 'reflect' reflects the shifts in xs, rather than shifting x's reflections.)
• analytic: should be false to prioritize effective temporal compactness of wavelets; the accurate A.M. mapping achieved by true isn't relevant here.
• max_pad_factor: should be unrestricted for best results.
• precision: should be double, as these experiments are highly numerically sensitive.

• out_exclude: do only spinned coefficients, as we focus on the FDTS-capturing part of JTFS. We can also do this in practice, by emphasizing spinned coefficients or excluding others.
• sampling_filters_fr: compute all spinned coefficients, without exclusions, for completeness.

• Left: signal, zoomed
• Right: (modulus of) CWT of signal
• Bottom-top: TS of signal, using T = signal duration
• Bottom-bottom: JTFS of signal (spinned coefficients only), using same T

• Lowering T lowers the ratio, as TS becomes more sensitive.
• Including all JTFS coefficients (i.e. non-spinned also) lowers the ratio, but the ratio stays high. That's because, by design, only spinned coefficients are FDTS-sensitive. In practice, we can choose to emphasize spinned coefficients, or exclude non-spinned.

Frequency shift insensitivity

• F: we make multiple JTFS, each with its own F. For "F=0", we set F close but not equal to 0, since the library requires F>0. For "F='global'", refer to the explanation in the makePhiFrExactlyGlobalAveraging() helper.

• Top plots: left is signal, right is its scalogram.
• Bottom plots: top is TS, below it are JTFS with varying F.

• Unlike time-shifts, a shift in log-frequency is inexact. That is, the effect on the transform, in case of CWT, isn't actually ; in short, a frequency shift in CWT causes extension in time, since the wavelets are multi-resolution.
• For linear frequency shifts, i.e. shifts of form , where is the Fourier transform of x, there's still imposed insensitivity, but its extent is controlled in a (even more) non-straightforward manner.

• F=0, i.e. no frequential averaging, the ratio is lower than 1 due to frequential scattering, which smoothes along frequency.
• F=8, the ratio is significantly lower, and for F=32 it's yet lower, but both are considerable for the shift in our example.
• F='global', the ratio is very low, making JTFS practically shift-invariant. The ratio not being zero is consistent with the earlier note on CWT, but for most practical purposes, the difference is negligible.

Example: exponential chirp

• average, average_fr: both false, as per our objective.
• oversampling, oversampling_fr: set to maximum to disable subsampling, as we require equal dimensionalities for visualization.
• T, F: still apply to non-spinned pairs, so we match them against J and J_fr (consistent with standard idealized configuration (2)).
• paths_exclude: for a more compact visualization, keep number of coefficients limited.
• max_noncqt_fr: our "CWT" is a mix of CWT and approximate STFT; setting max_noncqt_fr=0 disables including any non-CWT coefficients into joint pairs, which aids with example's idealization.

• Vertical axis: xi1_fr, log-quefrency, in cycles/octave (scattering along log-frequency).
• Horizontal axis: xi2, log-frequency, in cycles/sample (second-order scattering along time).
• To convert xi2 to Hertz: (1) multiply by sampling rate, fs (e.g. if fs=1000 Hz, then xi2=0.0125 corresponds to 1000*0.0125=12.5 Hz); (2) pass fs to viz_jtfs_2d in make_gif_echirp2d.

Disentangling variability

JTFS linearizes and disentangles core sources of signal variability: carrier frequency (frequency offset), modulation frequency (A.M.), and chirp rate γ(F.M.). We reproduce and explain this result from [2].

Denoting the triplet with , we have

,

• is the carrier's instantaneous frequency. (Its derivative is the argument to the sine of carrier, since the integral of the argument is the frequency.)
• γ is in octaves per second, measuring the rate of change of frequency. An "octave" is a doubling in frequency. For , the frequency doubles over two seconds.
• , the (amplitude) modulator's frequency, is constant.
• is a Tukey window with effective support of , where w is number of octaves. It's such that, w octaves are spanned by the effective support. (E.g. if octaves/second and octaves, it'll take seconds to sweep two octaves.)

To make coefficients more comparable across different , we use a fixed w, making coefficients have the same effective (frequency) span, so the only differences are due to energies and (frequency) offsets of coefficients. We use . Note, some implementation specifics, and value of w, differ from that of [2].

Visualize a few :

• First, second, third rows: x zoomed, x more zoomed, zoomed.
• x amplitudes (particularly in last column) differ since the x are normalized to have same energy (not shown in the expression for ).
• Row 3, plot 1: vertical center is at 512 Hz, consistent with . The effective duration is about 2 seconds, matching . We count 8 modulation cycles (including fading ones), consistent with .
• Row 3, plot 2: vertical center is at 1024 Hz, all else same, consistent with this .
• Row 3, plot 3: vertical center is unchanged, slope (chirp rate) is unchanged, duration is unchanged, only modulation count (and rate) changes, consistent with this .
• Row 3, plot 4: vertical center is unchanged; slope is increased, and to satisfy , duration is decreased. Since duration is decreased, there are fewer modulations in total, but the modulation rate (per unit time) is same.

• Large black dots = reference points.
• Blue solid line = geodesic path between the points.
• Blue dashed line = Euclidean ("straight line") path between the points.
• Red line = neighborhood graph constructed by Isomap, using n_neighbors = 7.

We build Isomaps for TS and JTFS, sweeping logarithmically with sweep size 16, forming a total of 4096 signals. With , we obtain the following Isomaps:

• A point = transform of an x for a given .
• Blue = low , red = high , white = in-between.
• Points closer together are transforms of whose geodesics are closer (see below). That is, closer points = more similar transforms.

For TS, we see blue and red points packed close together. This means that signals with significantly different have similar time scatterings, which is undesired. Meanwhile, JTFS's isomap is 3-D, matching that of our 3-D parameter space, and points are distributed approximately linearly (in a cube shape) and uniformly, maintaining more even gaps for differences in parameter space.

TS reliably maps and γ (see [2]), forming a 2-D manifold, but fails with . The manifold being 2-D makes it incapable of mapping distances consistent with the parameter space, which is 3-D. If we nudge or γ, we nudge along one of the two dimensions in TS's 2-D manifold; if we nudge , it will also nudge along one of the two dimensions, making it entangled with the other parameters. JTFS disentangles all three parameters.

Why this difference? We know from [2] that TS doesn't struggle with (which lacks frequency resolution), so we hypothesize that higher Q degrades the similarity retrieval of amplitude modulations. What is the dependence, and is there a "tipping point" Q? We investigate by visualizing:

TS's 3-D Isomaps gradually collapse into 2-D along the dimension. While there isn't a strict point at which the collapse is "finalized", the manifold is visually flat past around . Meanwhile, JTFS retains its locally and globally quasi-linear distribution.

We observe, that the individual rows are highly similar in shapes. Following the GIF sweeping to , we see this is per progressively degrading time resolution, temporally expanding narrow modulation cycles. At the same time, the energies of columns (sum of squares along time slices) encode substantially different modulation rates. Indeed, while the overall first-order (unaveraged) transform fully encodes , it does so over multiple coefficients (rows); as TS operates on per-coefficient basis, it is agnostic to adjacent coefficients, and this spread-out encoding is not reflected at the second order. JTFS, on the other hand, operates jointly in time and frequency. Above the Isomap plots, we see the TS's coefficients are nearly same, while JTFS's have definite differences.

In this case, the transforms perform comparably. To show a difference, the entangled dimensions must be explored. As both transforms are frequency shift equivariant, both perform well with ; indeed, the well-separated "segments" for TS isomaps are due to . This leaves γ as the entangled dimension. As the transform with an extra degree of freedom and explicit sensitivity for FDTS, JTFS excels at mapping out γ, while TS relies on "surrogate" variability in individual coefficients. As we've seen earlier, JTFS is sensitive to substantial parameter differences, while TS can conflate them.

To illustrate the problem, we fix and γ, for two choices of γ: one where TS works well, and one where it doesn't. And, we sweep .

Let be the initial . We observe from coefficient differences, and relative Euclidean distances, that for one of the γ, both JTFS and TS consistently grow away from . For the other γ, however, while JTFS behaves as before, TS first quickly grows away from , then grows closer to it, despite still growing away from . This shows that TS distances do not closely correspond to parameter distances in .

To show that and γ are entangled, we consider what happens for a neighboring choice of γ relative to the poorly-performing γ. Suppose this neighbor is well-behaved, and distances always grow with growing . Since this isn't the case for the reference γ, however, points close to in the neighbor γ can be about equally close to in the reference γ.

This means that the points cannot be distributed along separate, equally-spaced 1D manifolds. One candidate, with just two γ considered, is that there are two 1D manifolds that share a point (e.g. ) and are angled with respect to it, such that additional γ would form a circle. As differing γ generates a distance (so many γ with same cannot intersect), this configuration doesn't occur; instead, the 1D manifolds, while separated, are tightly and irregularly packed, and sometimes (approximately) intersect. Hence, the 1D manifolds do not group into a well-behaved extra dimension, and there is entanglement: points with different or different γ can be close, and incrementing either doesn't consistently increment distances.

To visualize these observations, we compare a Isomap slices with our fixed :

TS Isomap is compressed vertically (which, as we'll see, is the dimension), and points within it are irregularly spaced and irregularly ordered. Meanwhile, JTFS Isomap is roughly uniform both vertically and horizontally.

We see that for different γ, JTFS is distributed along two distinct, roughly equally spaced 1D manifolds, while TS is irregularly spread out in 2D. JTFS is well-ordered, while TS has instances of points such that, farther are closer than closer (blue points close to red points).

Disentangling variability (extended)

Time shift insensitivity

Wavelet scattering's signature property, insensitivity to time shifts, , also carries to JTFS. To illustrate, we

1. Tukey-window White Gaussian Noise of 4096 samples.
2. Shift it by 256 samples.
3. Apply TS and JTFS to the original and its shifting, using same T for both. For JTFS, apply it using multiple F, from 0 to 'global'.
4. Compute relative distances between transforms of shifted and unshifted signals.
5. Repeat 3-4 for all T, from 0 to 'global', swept in powers of 2 (0, 1, 2, 4, 8, ...).

• out_3D=true minimizes information losses due to unpadding by unpadding more conservatively, yielding more accurate measurements (4).
• For F=0, we instead set F close to machine epsilon, to avoid internal numeric errors; the difference in outputs is negligible.

• Recall the echirp earlier, and imagine shifting it by a small amount horizontally (i.e. in time), until it no longer overlaps itself.
• Now imagine lowpass filtering it vertically (i.e. along frequency). The scalogram is blurred, or "widened", vertically, and this additional support makes it overlap with its shift again, lowering distance.
• With global averaging, F='global', where the scalogram completely stretched out vertically (and can be collapsed to 1D, shaped [1 N]), this effect is maximized, and most of the scalogram overlaps its shifted self.

Warp-stability, noise-robustness

• Warp-stability: a time warp is a deformation of form , where ( implies time stopping, and time reversal; is a pure shift and is included). TS is proven to be stable for up to in [8] (also see (6)). "Stable" in context means, in simple terms, that the distance between transforms of and is bound by the "size" of deformation, not characteristics of x.
• Noise-robustness: TS is stable against additive noise. This follows its contractiveness property: , where S is scattering [9]. If , it says that distance between scattering of x and noisy x cannot exceed the distance between x and noisy x.

Applications of JTFS properties

1. Audio classification: strong performance was shown in a wide variety of tasks [2] [3] [4] [5].
2. Biomedical classification: strong performance was shown in [10], but existing research is limited.
3. Audio synthesis: advantages over TS for audio texture synthesis, in better preserving alignment of partials upon high T, are shown in [2], [3]. [6] showed how spinned coefficients can be used to locally time-reverse a signal. [2], [5] showed that JTFS accurately models auditory similarities, and [7] additionally showed how JTFS, unlike the spectrogram or scalogram, is viable for mesoscale parameter retrieval.

• FDTS discriminability: taking an audio signal with many transients, such as a jackhammer in a street scene, and misaligning its subbands yields a completely different sound [3]. Seismic signals can exhibit frequency-dependent dispersion characteristics, which can be used to assess changes in the structural integrity of a medium, or to analyze the Earth's structure. Different mechanical faults (e.g. bearing wear, gear damage, shaft misalignment) alter a system's stiffness and damping, causing FDTS in vibration signals; FDTS insensitivity would hinder fault diagnosis.
• Frequency-shift insensitivity: speaker identification is sensitive to pitch, but speech recognition in non-tonal languages isn't [3]. Same instruments can be played at different pitches, insensitivity hence aids musical instrument recognition. As ECG can vary by a person's baseline heartrate, insensitivity aids with recognizing arrhythmia patterns across patients.
• FDTS sensitivity: crucial to JTFS's generality is that "FDTS" need not be some explicit transformation. JTFS doesn't know, nor cares, whether this occurred. Only variation in time-frequency is needed, which is true of anything that's not a pure sine. The utility of JTFS lies in preserving frequency modulation information against time averaging using much fewer scattering orders than TS. While both can discriminate a unit impulse from an exponential chirp, TS requires many more orders and coefficients than JTFS to do so with a high T. Another utility is in having a richer decomposition: the added degree of freedom via frequential scattering introduces "quefrency", which by itself separates F.M. and A.M. more granularly than TS by shape and size of variation, but also together with second-order "frequency", adds the "slope" descriptor, decomposing F.M.s by their rates of change. While explicit FDTS phenomena are limited, frequency modulations are nearly universal.
• Time-shift insensitivity, time-warp stability, noise-robustness: foundational scattering properties which JTFS inherits. In synthesis, shift-insensitivity benefits parameter retrieval (e.g. [7]), as the input recording can be arbitrarily shifted without affecting parameters of interest. In classification, classes are ubiquitously invariant to time shifts; delayed trumpet or tachycardia are still trumpet and tachycardia. General or further discussion of benefits of shift-insensitivity, or of warp-stability or noise-robustness can be found in scattering literature (e.g. [9]).

JTFS vs TS: when to use which?

• JTFS provides more "knobs" to turn (kinds of coefficient manipulations to use), enabling finer control. That is, unless, we pursue third- or higher-order TS, where the "knobs" between the transforms differ. However, JTFS can simply be paired with higher orders of TS.
• JTFS's preserving of FDTS sensitivity through increased time-shift insensitivity can also be beneficial, where said insensitivity is needed.
• TS is undoubtedly faster, and can be sufficient if JTFS's added properties aren't relevant.

• FDTS not relevant: "... this increased discriminative power may not always be desirable. For example, frequency-dependent time-shifts ... or similar transformations may not be relevant for the classification task. In this case, replacing the time scattering transform with a joint time-frequency scattering transform would needlessly increase the number of model parameters, potentially requiring more training data to train an accurate classifier." [3]
• T is small: "For phone identification, we therefore require the invariance scale T to be of this order. Since T is small, there is less room for the type of misalignment seen in Section III-A. We therefore expect the joint time-frequency scattering to provide only limited improvement over time scattering." [3]

• JTFS beats TS, almost always. For numerous reasons beyond the scope of this example, JTFS can be considered as strictly superior to TS for classification. A part of reasoning concerns the description in previous section on "FDTS sensitivity". If a real-world task requires an automated classifier, odds are extremely low that it's comprised mainly of stationary signals without frequency varying over time. Moreover, JTFS can be paired with higher-order TS, if loss of information due to high T is a concern. Size of transform (hence dimensionality) is controllable, but the idea is, it's a price worth paying. With absence of extensive ablation studies, "almost always" is the author's bet.
• Compute-efficiency: despite having superior performance, JTFS's performance-to-compute ratio can be worse. A priori determination of said ratio isn't practical. However, with "amount of time-varying frequency" (relative to stationary contents) mattering most, one can define a heuristic to measure it, and iterate over the dataset. While outside the scope of this example, we provide a summary in (7).
• T is small: if small T is preferred for whatever reason, then TS is likelier to outperform JTFS, as latter is overly redundant and high-dimensional (in agreement with "mainstream"). Compensating for reduncancy with large F isn't generally safe, as the frequency axis holds a lot of important information. However, it should be rare that for reasons other than compute efficiency, a small T is preferred over JTFS + higher order TS with high T, even if task-relevant structures are narrowly localized in time.
• Simplicity: designing an appropriate learning algorithm to go on top of JTFS (e.g. convolutional networks) is more challenging than for TS. Also, TS generally involves less paramter tuning. For appropriate tasks, JTFS may nonetheless work better out-out-box or require less tuning: the parameters that matter most are T, Q, and F if average_fr=true. Worth trying are Q=8,16,24 and T=N/8,N/32 as starters.

Extended notes

(1) Strictly speaking, it can, but it would take a great number of orders, more than in virtually any application. TS is only invariant to two kinds of transformations: a global phase shift, and a global time shift if T='global'. Yet, at the same time, due to its tree structure, TS cannot distinguish between and any arbitrary global time-shifting of individual its frequencies λ, including if said shifts are different. What keeps TS discriminative is the fact that such transformations are impossible: there aren't x that yield such X. This includes the exponential chirp formed from a unit impulse: such a chirp would need to be correlated with each frequency λ at only one time instant, which is impossible. The power of JTFS lies in achieving this discriminability with only two orders.

(3) Strictly speaking, it is variation in time and log-frequency slices of , over time, which does happen, due to the wavelets varying in time support and bandwidth as a function of xi1. This variation is small for small sweeps across xi1.

(6) While [9] states the result for , [8] proves it only for . Note, "size" of deformation includes and , respectively, the gradient of deformation, and size of deformation (with J dependence). [8] states that can be replaced with if the additional dependence of is introduced to the bounding distance. Yet, as , this term explodes, making the bound non-existent. For not too close to 1, we still see some approximate stability, but the scattering distance bound is no longer linearly proportional to size of deformation.

(8) Any T='global' isn't affected at all, as there isn't unpadding. The frequency transposition example is negligibly affected. In the time-shift insensitivity example, the difference in plots is perceptible, but small. (The explanation for the two aforementioned cases concerns effective signal support given windowing, hence the amount of unpad losses, and whether time shifting is involved, which also affects losses.) The echirp example's ratio does change substantially (in testing, from to ), which is as expected as the boundary effect is exacerbated (more of it is kept), but a more realistic chirp example (and incidentally, more idealized) involves windowing, which'd greatly reduce this effect; windowing was omitted to illustrate this effect.

References